Scheduler in multi-threaded processor prioritizing instructions passing qualification rule

ABSTRACT

A processor buffers asynchronous threads. Instructions requiring operations provided by a plurality of execution units are divided into phases, each phase having at least one computation operation and at least one memory access operation. Instructions within each phase are qualified and prioritized. The instructions may be qualified based on the status of the execution unit needed to execute one or more of the current instructions. The instructions may also be qualified based on an age of each instruction, status of the execution units, a divergence potential, locality, thread diversity, and resource requirements. Qualified instructions may be prioritized based on execution units needed to execute instructions and the execution units in use. One or more of the prioritized instructions is issued per cycle to the plurality of execution units.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationtitled, “MULTI-THREADED PROCESSOR APPARATUS AND METHOD,” filed Apr. 13,2006, and Ser. No. 11/404,196, which is a continuation-in-part ofco-pending U.S. patent application titled, “MULTI-THREADED PROCESSORAPPARATUS AND METHOD,” filed Nov. 17, 2004, and Ser. No. 10/991,640. Thesubject matter of each of the aforementioned non-provisional patentapplications is hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention is generally related to multi-threaded processors.More particular, the present invention is directed towardsmulti-threaded processors having dedicated execution units for executingthread instructions.

2. Description of the Related Art

Multi-threaded processors are of increasing interest in a variety ofapplications. A multi-threaded processor has multiple threads forprocessing information. For example, multi-threaded processors are ofinterest for use in Graphics Processing Units (GPUs).

A GPU commonly includes stages dedicated to performing specifiedfunctions. An emerging problem is designing a multi-threaded GPUarchitecture that efficiently utilizes GPU resources, such as executionpipelines.

Therefore, what is desired is an improved multi-threaded processorarchitecture, and a new method and apparatus for qualifying andprioritizing instructions for execution by the threads.

SUMMARY OF THE INVENTION

A multithreaded processor buffers current instructions of threads andprioritizes the issue of current instructions to execution units. Oneembodiment of a processor includes a plurality of different types ofexecution units, each type of execution unit servicing a different classof operations. In one embodiment there is one or more of each type ofexecution unit. An instruction buffer buffers a set of currentinstructions for a plurality of asynchronous threads, each currentinstruction requiring an operation performed by one of the plurality ofdifferent types of execution units. An instruction scheduler qualifiesat least a portion of the current instructions for execution andprioritizes the qualified instructions. The instruction scheduler issueson each issue cycle one or more of the prioritized instructions from theinstruction buffer to the plurality of different types of executionunits.

Various embodiments of the current invention provide a multi-threadedprocessor generally including a plurality of different types ofexecution units, an instruction buffer, and an instruction schedulerthat includes a qualification module and a prioritization module. Atleast one of each type of execution unit is configured to service adifferent class of operations. The instruction buffer is configured tobuffer a set of current instructions for a plurality of asynchronousthreads, each current instruction requiring an operation performed byone of each of said plurality of different types of execution units. Thequalification module is configured to qualify at least a portion of saidset of current instructions for execution based on a qualification ruleto produce a set of qualified instructions. The prioritization module isconfigured to prioritize said set of qualified instructions based on aprioritization rule to produce prioritized instructions for issue fromsaid instruction buffer to said plurality of different types ofexecution units

Various embodiments of a method of the invention for operating amulti-threaded processor include buffering current instructions forasynchronous threads, a current instruction of an individual threadrequiring one of a plurality of different types of execution units toservice a class of operations, qualifying buffered current instructionsto produce a set of qualified instructions, prioritizing said qualifiedinstructions to produce prioritized instructions based on rules toimprove utilization of said different types of execution units, and foreach issue cycle, issuing at least a portion of said prioritizedinstructions to said plurality of execution units for execution.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 is a block diagram of a multi-threaded processor in accordancewith one embodiment of the present invention;

FIG. 2 is a flow chart of a method of issuing instructions to executionunits in accordance with one embodiment of the present invention;

FIG. 3 is a block diagram of a multi-threaded graphics processing unitin accordance with one embodiment of the present invention.

FIG. 4A is a flow chart of a method of performing step 210 of FIG. 2 inaccordance with one embodiment of the present invention;

FIGS. 4B and 4C are flow charts of method of performing step 400 of FIG.4A in accordance with one embodiment of the present invention;

FIG. 5A is a flow chart of a method of performing step 402 of FIG. 4A inaccordance with one embodiment of the present invention;

FIG. 5B is a flow chart of a method of computing issue-credit counts foruse in step 402 of FIG. 4A in accordance with one embodiment of thepresent invention;

FIG. 5C is a flow charts of method of performing step 524 of FIG. 5B inaccordance with one embodiment of the present invention; and

Like reference numerals refer to corresponding parts throughout theseveral views of the drawings.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the present invention. However,it will be apparent to one of skill in the art that the presentinvention may be practiced without one or more of these specificdetails. In other instances, well-known features have not been describedin order to avoid obscuring the present invention.

FIG. 1 is a block diagram illustrating a multi-threaded processor 100 inaccordance with one embodiment of the present invention. Multi-threadedprocessor 100 includes a thread pool 105. In one embodiment, the threadsin thread pool 105 run asynchronously, i.e., they are not synchronizedwith each other. Consequentially, at any one particular time, individualthreads in the thread pool of threads 105 may require differentoperations.

In one embodiment, thread pool 105 includes different thread types forprocessing different types of information. The threads may include aninteger number of different thread types, such as thread type 1, threadtype 2, . . . thread type K, where there are a plurality of threads ofeach type within thread pool 105. A particular thread type may, forexample, be a computing thread type or a graphics thread type. Forexample, in a Graphics Processing Unit (GPU) implementation, thread pool105 may include vertex threads for processing vertex information andpixel threads for processing pixel information. In some implementations,different types of threads may request different types of operations.However, more generally, different types of threads may request at leastone operation that is identical. For example, in some embodiments bothpixel threads and vertex threads may request the same type ofmathematical operation to be performed.

An instruction memory 110 fetches a current instruction for each threadbased on the program counter of the thread. Instruction memory 110 may,for example, have a multi-level structure with one or more local caches(not shown) to improve efficiency.

An instruction buffer 120 associated with instruction memory 110 buffersa set of current instructions 125 for a set of threads. At any oneparticular time, instruction buffer 120 will have a certain number ofcurrent instructions 125 in instruction buffer 120. The number at aparticular point in time depends, for example on a fetch policy used toselect a new set of instructions to enter instruction buffer 120 and theissue policy used to issue instructions out of instruction buffer 120during instruction cycles.

In one embodiment a thread tagger 134 includes logic for tagginginstructions that are fetched into instruction buffer 120. The fetchedinstructions are tagged to indicate the type of execution unit 140required to execute the instruction.

Each current instruction in buffer 120 has one or more attributes, suchas a thread type attribute and an instruction age attribute. The threadtype attribute describes the thread type associated with theinstruction. The instruction age attribute may, for example, be a threadattribute related to age, such as a launch order or the age of theinstruction within instruction buffer 120.

Each of the instructions in instruction buffer 120 utilizes theresources of an execution unit 140 to perform an operation required bythe corresponding thread. In one embodiment, each execution unit 140-1 .. . 140-N is implemented in hardware as a unit that is configured toperform a class of operations for the threads. The class of operationsmay include a single operation but more generally may comprise a numberof different operations that execution unit 140 can perform. While eachexecution unit 140 may service a different class of operations, in oneembodiment there is more than one execution unit 140 to service at leastone class of operations.

Each execution unit 140 has an input for receiving a current instruction(not shown) and an associated latency for performing an operation for athread, e.g., an operation will take a number of cycles to complete. Inone embodiment, there is a positive integer number, N, of differenttypes of execution units 140, such as execution units 140-1, 140-2, . .. 140-N and each type of execution unit 140 is adapted to service adifferent class of operations for the current instructions of thethreads. For example, in the context of a graphics processing unit (GPU)implementation, execution unit 140-1 may be dedicated to servicinginstructions for texture operations, execution unit 140-2 may bededicated to servicing instructions for multiply-add (MAD) operations(e.g., blending operations), and execution unit 140-N may be dedicatedto servicing instructions for specialized mathematical operations, suchas the reciprocal, logarithmic, exponential, or other mathematicalfunction. In one embodiment each execution unit 140 is an executionpipeline, such as a programmable pipeline having a plurality of stages.

The execution units 140 each have operational attributes. For example,an individual execution unit 140 may be non-stallable or may be capableof stalling. At a particular point in time an individual execution unit140 may be in a normal or a stalled state. In an execution pipelineembodiment, the instructions are processed in stages, such that a newinstruction can enter a first stage of the execution pipeline once aprevious instruction has passed on to a subsequent stage.

An instruction scheduler 130 includes a qualification module 132 forqualifying valid instructions for issue to each execution unit 140.Additionally instruction scheduler 130 includes a prioritization module136 for prioritizing the issue order of instructions. As described belowin more detail, qualification module 132 and prioritization module 136permit an optimization of the selection of instructions for issuance tothe execution units 140 that improves execution unit utilization.

Instruction scheduler 130 selects a set of instructions per issue cycleto be issued to the execution units 140 via a bus 150. In oneembodiment, the maximum number of instructions that may be issued percycle is an integer number M, where M≦N. For example, in a processorwith a total of three different execution units 140-1, 140-2, and 140-N,the number of instructions issued per cycle may be 0, 1, 2, or 3. Themaximum number, M, may be limited due to the limitations of bus 150 orother constraints. Thus, in some implementations M<N, such that only asubset of the execution units 140 can receive an instruction in aparticular cycle. However, it will be understood that in an alternateembodiment that threads are grouped into convoys such that M>N. As anillustrative example, threads could be grouped into a group of ten andone group of ten instructions issued into each execution unit 140 every10th cycle.

Additionally, it will be understood that the instruction buffer 120 mayalso issue instructions to individual execution units 140 as a group. Inone embodiment, threads are grouped into pairs that share the sameinstruction. The single instruction that the pair of threads shares issent across bus 150. The execution unit that receives the instructionthen repeats the instruction twice, once for each thread. Thisimplementation has the benefit that the bus width does not have to bemodified, i.e., a single-width bus may be utilized. However, it will beunderstood that in an alternate embodiment, bus 150 is a double-widthbus adapted for instruction buffer 120 to fetch a pair of instructionsfor an execution unit 140 in each issue cycle, i.e., a pair ofinstructions is fetched once, then executed twice in succession by theexecution unit. In this embodiment, there is a maximum number of pairsof instructions that may be issued, where M≦N/2.

A register file 145 is provided for threads to perform register readsand writes. In one embodiment, register file 145 includes a bank ofindependent registers.

In one embodiment a scoreboard (not shown) is also included forinstruction scheduler 130 to track the status of instructions executingin each execution unit 140. In some embodiments, a monitor 155 isprovided to monitor the status of execution units 140-1, 140-2, . . .140-N for instruction buffer 120. Monitor 155 may, for example, detectstalls of individual execution units 140, collisions regardingwritebacks of one or more execution units 140 to register file 145, ortrack the status of operations in execution units 140.

Instruction scheduler 130 prioritizes the buffered instructions 125 todetermine a priority for issuing instructions. For each cycle,instruction scheduler 130 selects a number of buffered instructions 125to be issued to execution units 140-1 . . . 140-N. The instructionscheduler 130 issues a maximum number, M, of instructions per issuecycle that is less than the buffer capacity of instruction buffer 120.

Instruction scheduler 130 can select an issue order that is moreefficient than that which would occur if thread pool 105 directlyaccessed the execution units 140 without buffering and prioritization.This is because the asynchronous nature of the threads makes it likelythat the current instructions of different threads request differentoperations. Consequently, it is likely that work can be found for all ofthe execution units 140 if a sufficient number of current instructionsare buffered. Increasing the capacity of instruction buffer 120increases the statistical likelihood that instruction scheduler 130 canperform an optimization of the instruction issue order. Also, increasingthe capacity of instruction buffer 120 permits different types ofoptimizations. For example, increasing the capacity of instructionbuffer 120 increases the likelihood that current instructions 125include a distribution of thread type attributes, instruction ageattributes, or other attributes, thereby increasing the choices thatinstruction scheduler 130 can make to optimize the issue order. Forexample, the instruction buffer 120 may be sized to make itstatistically likely that when instruction buffer 120 is full theninstruction scheduler 130 can find a pre-selected number of instructions(e.g., at least one) that can be issued to each of the execution units140. In one embodiment, the instruction buffer is sized so that eachthread has one guaranteed slot in the instruction buffer (e.g., if thereare 24 threads there are at least 24 slots such that each thread has oneguaranteed slot). It will be understood that one of ordinary skill inthe art could perform modeling or empirical investigations to determinethe buffer capacity required for a particular implementation ofprocessor 100 to improve the utilization efficiency of execution units140.

The prioritization can be carried out each time instruction buffer 120is refreshed with new instructions from thread pool 105. However, sincethe status of execution units can change, in some embodiments theprioritization is updated to adapt to any changes in the executionenvironment. In one embodiment, a rules engine 138 within instructionscheduler 130 may apply one or more rules to achieve a variety ofobjectives, such as distributing work between execution units 140 toimprove the utilization of the various execution units 140 (e.g., tokeep each execution unit 140 filled with instructions), avoiding thedevelopment of too large a backlog of instructions for a particularexecution unit 140, and avoiding unduly delaying the execution of thethreads. The particular set of rules that is applied by rules engine 138will depend upon the implementation of processor 100 and upon thecombination of objectives that are desired. In one embodiment, rulesengine 138 is programmable to permit a set of rules to be programmed.

One example of a rule that may be applied by instruction scheduler 130is a qualification rule. In one embodiment, instruction scheduler 130qualifies an instruction for issuance to a particular execution unitonly if issuance of the instruction on a particular cycle would notgenerate a deleterious condition. Instructions that would generate adeleterious condition are disqualified. Disqualifying instructionspermits a more efficient utilization of the execution units 140. Forexample, in one embodiment if a particular execution unit 140 isunavailable for at least one issue cycle (e.g., the execution unit isblocked, busy, or stalled such that is not immediately available toperform an operation), the instruction scheduler 130 disqualifiesinstructions requiring the unavailable execution unit from being issuedin that cycle. This permits, for example, issuing instructions toexecution units that are available for execution of instructions. Insome embodiments, information from monitor 155 is used by instructionscheduler 130 to disqualify instructions that would generate a collisionof threads during writeback.

Instruction scheduler 130 may also use priority rules to assign apriority to each of the qualified instructions in instruction buffer120. One example of a priority rule is that some thread types may beassigned a higher priority than other thread types. As an illustrativeexample, in a GPU implementation an instruction for a vertex thread maybe assigned a higher priority than an instruction for a pixel thread.

Another example of a priority rule is a rule for assigning an issueorder based upon instruction age, such as assigning a higher priority toolder instructions in instruction buffer 120. For example, in oneembodiment older instructions for a particular thread type are assigneda higher priority than newer instructions. As previously described, theinstruction age may be any attributed related to age, such as a launchorder or other attribute.

Still another example of a priority rule is a round robin scheme inwhich buffered instructions are issued in a round robin order. As oneexample, the round robin may cycle around instructions for threads of aparticular thread type. More generally, any round robin scheme fordistributing work in a cycle may be utilized, such as a round robinorder based on cycling around the execution units.

Still yet another example of a priority rule is a rule for assigningpriority based upon instruction statistics. For example, in oneembodiment, priority is based upon the number of instructions requiringaccess to each execution unit. In one embodiment, the priority of aninstruction is increased as the number of other instructions requiringthe same execution unit increases. As an illustrative example, if thenumber of instructions that require a particular execution unit 140increases, preference may be given to issuing these instructions toreduce the backlog of such instructions.

As still yet another example of a priority rule, the priority rule maybe a rule selected to increase utilization of the execution units 140based upon program statistics regarding the likely sequence of threadtypes that will be fetched into instruction buffer 130. In particular,although the asynchronous threads are not synchronized, there may bestatistical patterns to the number and order in which threads ofdifferent types are fetched. For example, the statistical patterns mayinclude a burst behavior in which threads requiring a particularexecution unit tend to occur in bursts. One or more rules may be basedupon statistical patterns associated with a program to improveutilization of the execution units. For example, a particular programmay, for example, have a statistical likelihood that closely spacedasynchronous threads will require the same execution unit. In oneembodiment a spacing rule is applied that spaces out the execution ofthe threads to increase the statistical probability that a largepercentage of the execution units have work.

FIG. 2 is a flow chart of a method in accordance with one embodiment ofthe present invention. The instruction buffer is filled in step 205 withthe instructions of threads requiring execution. In one embodiment, oneor more rules are utilized by instruction buffer 120 for receiving newinstructions. For example, if instruction buffer 120 has a fixed size,the rule may be based on the number of current instructions 125 inbuffer 120. The priority of instructions in the buffer is assigned instep 210 based on thread attributes and execution unit attributes. Foreach cycle, a number of high-priority instructions are issued in step215 to the execution units.

FIG. 3 is a block diagram of an exemplary GPU implementation. Many ofthe elements perform similar functions as those described in regards toprocessor 100. In a multithreaded GPU 300 there is a pool of graphicsthreads 305, such as pixel threads or vertex threads. Instruction memory310 may comprise a level 1 (L1) cache coupled to a level 2 (L2) 312cache. If there is a cache miss, a graphics memory (e.g., a framebuffer) may be accessed for an instruction. In this embodiment, aninstruction buffer 320 buffers instructions of graphics threads 325 andincludes an instruction scheduler 330. Execution pipelines 340 maycomprise any execution pipeline used in a GPU for servicing graphicsthreads. A register file 345 is provided for register writes and amonitor 355 may be included in some embodiments for monitoring thestatus of execution pipelines 340.

As previously described, instruction scheduler 130 includesqualification module 132, prioritization module 136, and rules engine138. Qualification module 132 qualifies valid instructions for issue toeach execution unit 140 and prioritization module 136 prioritizes thequalified instructions into an issue order that may improve executionunit 140 utilization. Rules engine 138 may be programmed withqualification and/or prioritization rules that are applied byqualification module 132 and prioritization module 136, respectively.The rules may specify particular behavior based on the thread attributesand execution unit 140 operational attributes.

FIG. 4A is a flow chart of a method of performing step 210 of FIG. 2, inaccordance with one embodiment of the present invention. In step 400qualification module 132 produces a set of qualified instructions usingone or more qualification rules, as described in conjunction with FIGS.4B and 4C. In step 402 prioritization module 136 determines aprioritized order of the instructions in the set of qualifiedinstructions using one or more prioritization rules. Prioritizationmodule 136 produces a prioritized set of qualified instructions forissue to one or more execution units 140.

FIG. 4B is a flow chart of method of performing step 400 of FIG. 4A, inaccordance with one embodiment of the present invention. The steps shownin this flow chart are performed on each thread corresponding to thecurrent instructions 125 in buffer 120. Qualification module 132 mayreceive one or more rules from rule engine 138 that are used inconjunction with thread attributes and/or execution unit 140 operationalattributes to qualify or disqualify the current instruction for a threadbased on data integrity requirements, resource limitations, andperformance. Thread attributes may include scoreboard dependency bits,instruction age, issue cycle count, data cache hit status, serializationrequirements, or the like. Operational attributes may include whether ornot the execution unit may stall, storage resources, latency, the classof operations the execution unit may perform, and the like.

In step 410 qualification module 132 determines if the scoreboarddependency bits for the thread are clear, indicating that resourcesneeded to perform the current instruction for the thread are available.Resources may include a local register file, output buffers, addressregisters, condition code registers, or the like. If, in step 410qualification module 132 determines that the scoreboard dependency bitsfor the thread are not clear, qualification module 132 proceeds directlyto step 424. Otherwise, in step 412 qualification module 132 determinesif there is an older instruction for the thread in buffer 120, and, ifso, qualification module 132 proceeds directly to step 424. Otherwise,in step 414 qualification module 132 determines if an issue cyclecounter for the thread has reached a predetermined minimum value.

The issue cycle counter for a thread is reset whenever an instruction isissued for the thread and increments for every following issue cyclewhen an instruction is not issued for the thread. It may be used tocontrol the frequency at which threads issue instructions. For example,in some embodiments of the present invention, instructions for the samethread may only issue once every 4 issue cycles. This feature may beused to accommodate latency of feedback signals. If, in step 414qualification module 132 determines that the issue cycle count for thethread has not reached the predetermined minimum value, thenqualification module 132 proceeds directly to step 424. Otherwise,qualification module 132 continues to step 416. In step 416qualification module 132 determines if an in-flight instruction for thethread, i.e., instruction that has been issued but not completedexecution, is waiting for a hit in the data cache, and, if so,qualification module 132 proceeds directly to step 424. Informationregarding data cache hits may be provided to qualification module 132 byexecution unit status monitor 155. This feature may be used toaccommodate re-issuing the instruction that caused the data cache miss.

If, in step 416 qualification module 132 determines that an in-flightinstruction for the thread is not waiting for a hit in the data cache,then in step 418 qualification module 132 determines if an in-flightinstruction for the thread may require serialization, and, if so,qualification module 132 proceeds directly to step 424. Serializationmay be needed due to operand divergence, in which case the instructionis reissued as many times as needed to serialize the data accesses.Otherwise, qualification module 132 proceeds to step 420 and determinesif a synchronization state exists for the thread. A synchronizationstate may result from execution of a barrier instruction to synchronizetwo or more threads or to complete execution of an in-flight instructionfor the thread. If, in step 420 qualification module 132 proceeds tostep 420 and determines that a synchronization state exists for thethread, qualification module 132 proceeds to step 424. Otherwise, instep 426 qualification module 132 qualifies the current instruction(s)for the thread for execution and the instructions are included in theset of qualified instructions. In step 424 qualification module 132disqualifies the current instruction(s) for the thread for execution andthe instructions are not included in the set of qualified instructions.Disqualified instructions remain in buffer 120 until they are qualifiedand issued.

FIG. 4C is a flow chart of another method of performing step 400 of FIG.4A, in accordance with one embodiment of the present invention. Thesteps shown in this flow chart are performed on all of the threads thatcorrespond to the current instructions 125 in buffer 120. As previouslydescribed, qualification module 132 may receive one or more rules fromrule engine 138 that are used in conjunction with thread attributes toqualify or disqualify the current instruction for the threads. The rulesused in the steps shown in FIG. 4C may disqualify or qualify currentinstructions based on hardware limitations. One or more of the stepsshown in FIG. 4B may be combined with one or more of the steps shown inFIG. 4C to perform step 400 of FIG. 4A.

In step 430 qualification module 132 determines if a storage resourcelimit exists for the threads, and, if so, then qualification module 132proceeds directly to step 436. A storage resource limit may exist forparticular operations, such as memory read or write requests or loadingdestination registers. If, in step 430 qualification module 132determines that a storage resource limit does not exist, then in step432 qualification module 132 determines if a resource sharing limitexists for the threads, and, if so, then qualification module 132proceeds to step 436. A resource sharing limit may exist when anin-flight instruction needs a resource for multiple clock cycles. If, instep 432 qualification module 132 determines that a resource sharinglimit does not exist for the threads, then in step 434 qualificationmodule 132 determines if a branch limit exists for the threads. If, instep 434 qualification module 132 determines that a branch limit doesnot exist for the threads, then in step 438 qualification module 132includes the current instructions for the threads in the set ofqualified instructions. Otherwise, in step 436 qualification module 132disqualifies the instructions. A branch limit may exist because branchinstructions may not be issued in consecutive issue cycles.

In some embodiments of the present invention, an instruction may specifya preference for execution by a particular type of execution unit 140when more than one type of execution can execute the instruction. Aqualification rule may be used to configure qualification module 132 toignore the specified preference when another execution unit 140 isavailable to execute the instruction. Another qualification rule may beused to configure qualification module 132 to qualify the instructiononly if the specified execution unit 140 is available. In otherembodiments of the present invention, additional steps may be includedin the flow charts shown in FIGS. 4B and 4C to implement otherqualification rules.

FIG. 5A is a flow chart of a method of performing step 402 of FIG. 4A inaccordance with one embodiment of the present invention to produce a setof prioritized instructions. The steps shown in this flow chart areperformed on the set of qualified threads to optimize the instructionissue order for efficient utilization of execution units 140.Prioritization module 136 may receive one or more rules from rule engine138 that are used in conjunction with thread attributes and/or executionunit 140 operational attributes to prioritize the current instructionsin the set of qualified instructions for improved performance. Inaddition to the previously described thread attributes, threadattributes may also include position (in screen space), memory requestsegments, instruction type counts, thread type, thread issue count, orthe like.

In step 502 prioritization module 136 determines if there are any memoryrequest instructions that are included in the current phase of asegmented program, and, if not, prioritization module 136 proceeds tostep 506. A memory request instruction is an instruction that requires aread from or a write to memory during execution. For example, a texturefetch instruction requires a memory read. In order to improve cachecoherency or efficient memory access, a program may be segmented, i.e.,divided, into different phases to group memory request instructions andconstrain cache and/or memory access. As an example, a large executableprogram with many threads running may be subdivided in an effort to makemore efficient use of an instruction cache (I-cache) by enforcing adegree of locality and preventing different threads from evicting cachelines of instructions for other threads.

As another example, a shader program may be subdivided into phases ofexecutable instructions for processing samples, each phase having one ormore math operations and one or more texture cache access operations.The subdivision may be performed by a compiler that may surround eachtexture block including one or more texture memory accesses with severalmath operations, in an effort to ensure the shader is performing workwhile waiting for textures. These structures (block-texture, block-math)may be repeated throughout the executable code. The number of texturefetch operations in each texture block may, at least in part, be chosento ensure the number of textures fetched by all operations in a blockwill fit in a texture cache to prevent the operations from fighting forcache space. For example, if the texture cache is sized to hold threetextures, three texture fetch operations may be grouped in each textureblock. This approach may also lead to efficient utilization of thetexture cache. For example, as a first texture fetch operation iscompleting another texture fetch operation may begin. As a result, thelatency of the second texture fetch operation may be hidden as thesecond texture fetch operation is performed “in the shadow” of the firsttexture fetch operation.

During execution of the segmented shader program, instruction scheduler130 qualifies and prioritizes math and texture cache access operationsin a current phase of the shader program until a boundary between thecurrent phase and a subsequent phase is reached. Each instruction mayinclude a phase ID, inserted by the compiler during segmentation.Instruction scheduler 130 continues to qualify and prioritize currentinstructions until a phase boundary is reached, at which point texturecache access operations in the subsequent phase are not qualified untilall texture cache access operations in the current phase have completed.This allows the execution of texture operations in the subsequent phaseafter all texture cache access operations in the current phase havecompleted. In some embodiments of the present invention, math operationsin the subsequent phase may be qualified (based on a qualification rule)before all of the texture cache access operations in the current phasehave completed. Because math operations typically do not access thetexture cache, they may be allowed to execute somewhat randomly, withouthaving to wait for previous phase texture blocks to complete.

Returning to FIG. 5A, if, in step 502 prioritization module 136determines that there are memory request instructions that are includedin a segment, then in step 504 prioritization module 136 orders thememory request instructions based on the segment order. In step 506prioritization module 136 prioritizes all instructions in instructionbuffer 120 by instruction type as they pertain to pipeline resources. Insome embodiments of the present invention, counts may be computed forthe number of instructions that require each computation resource, eachstorage resource, and memory request type. The counts may be weighted toincrease or to decrease the priority of the corresponding instructiontype. For example, different types of memory requests may be weighteddifferently, giving some types of memory requests higher or lowerpriority relative to other types of memory requests. Instructions in thequalified set of instructions that match the type of memory request withthe highest count may be prioritized for issue ahead of otherinstructions by prioritization module 136.

In step 507 prioritization module 136 orders the current instructions inthe qualified set of instructions based on position to improve localitybetween the different threads. Improving locality may result in moreefficient cache accesses since data for nearby positions may be loadedinto the data cache when data for a first position is fetched.Therefore, cache thrashing may be reduced and the hit rate improved.

In step 508 prioritization module 136 orders the current instructions inthe qualified set of instructions based on thread type. For example,vertex threads may have priority over pixel threads. In step 510prioritization module 136 orders the current instructions in thequalified set of instructions by performing a search using one or morecounts maintained for each thread, as described in conjunction withFIGS. 5B and 5C. Prioritization module 136 may perform the instructionsearch starting with the thread corresponding to a qualified instructionthat is the oldest, based on the thread age attribute. Prioritizationmodule 136 may order the instructions from oldest to youngest as itsearches to produce the set of prioritized instructions. Prioritizationmodule 136 may maintain a counter corresponding to the thread age,incrementing the counter as the instructions are ordered. Differentcounters may be used for different thread types.

In another embodiment of the present invention, prioritization module136 may perform the search starting with the most recently issuedthread, i.e., the thread corresponding to the most recently issuedcurrent instruction, to perform a round-robin prioritization. In stillanother embodiment of the present invention, another count, anissue-credit count maintained by prioritization module 136 for eachthread, is used to prioritize the set of qualified instructions.Prioritization module 136 may order the instructions in order ofdecreasing issue priority as the issue-credit counts increase to producethe set of prioritized instructions. The method of computing theissue-credit counts is described in conjunction with FIG. 5B. Rulesengine 138 may specify the prioritization rule used by prioritizationmodule 136 in step 510.

FIG. 5B is a flow chart of a method of computing issue-credit counts foruse in step 402 of FIG. 4A, in accordance with one embodiment of thepresent invention. In step 520 all thread issue-credit counts areinitialized to zero. An issue-credit count is maintained for each threadin buffer 120. Step 520 may be completed when buffer 120 is loaded withcurrent instructions and the thread counts may be maintained byprioritization module 136. In step 522 prioritization module 136determines if a current instruction for one or more threads have issued,and, if not, prioritization module 136 proceeds directly to step 526.Otherwise, in step 524, the issue-credit count for each of the issuedthreads is incremented, as described in conjunction with FIG. 5C. Insome embodiments of the present invention, the issue-credit count may beincremented by more than one for particular instructions. For example,incrementing issue-credit counts for memory request instructions by morethan one may improve memory access locality.

Threads with the lowest count may be given the highest issue priority.Therefore, threads that have executed farther through the program willcontinue executing if and only if threads that have fallen behind areall stalled. Although the threads execute asynchronously, moresynchronous execution is encouraged using issue-credit counts.Furthermore, thread launch order, i.e., the order in which threads areallocated to execute a program, is preserved over longer time scales.

In step 526 prioritization module 136 determines if buffer 120 has beenloaded with a new program, and, if not, prioritization module 136returns to step 522 to update the issue-credit counts for the next issuecycle. Otherwise, in step 528 prioritization module 126 initializes theissue-credit counts of newly launched threads as a maximum issue-creditcount of the existing threads increased by one, then goes directly to522. The increase of one ensures that the existing threads have issuepriority compared with the newly launched threads. If, duringprioritization, prioritization module 136 determines that two or morequalified instructions have equal issue-credit counts, other attributes,such as thread age, may be used to order the two or more qualifiedinstructions relative to each other. For example, ordering schemesdescribed in conjunction with FIG. 5A may be used.

A benefit of using the issue-credit count to prioritize instructions isthat memory access efficiency may be increased while maintainingcomputational efficiency since threads are likely to issue in launchorder. For pixel threads, the threads are usually launched in rasterorder, therefore pixel locality is maintained by issuing pixel threadinstructions in launch order. Another benefit is that the issue-creditcount of stalled threads will increase, thereby increasing the priorityof stalled threads such that prioritization module 136 will favor thosethreads when they are in the set of qualified threads, i.e., when thestalled threads are no longer stalled. Therefore, threads that havefallen behind may catch up as their priority increases based on theirdecreasing (relative to the issue-credit counts of other threads)issue-credit counts. Threads may stall due to resource conflicts, memorylatency, resource limitations, or the like. Execution unit statusmonitor 155 provides instruction scheduler 130 with information onthreads that are stalled.

Overflow of an issue-credit counter may be handled in a variety of ways.In one embodiment of the present invention, the issue-credit counter maysaturate, when a maximum value that can be represented by the counter isreached, limiting the range over which thread separation may be reduced.When an issue-credit counter overflows an overflow bit is setcorresponding to that issue-credit counter. Saturation allows theissue-credit counts to continue to be used for prioritization, sincethreads with saturated issue-credit counters, i.e., set overflow bits,may be given lowest issue priority. All of the overflow bits are resetto zero once all of the issue-credit counters have overflowed in orderto maintain separation information between the threads.

FIG. 5C is a flow chart of another method of performing step 524 of FIG.5B, in accordance with one embodiment of the present invention. In step550 prioritization module 136 increments the issue-credit counter of theissued thread(s). In step 552 prioritization module 136 determines ifany incremented issue-credit counter is overflowed, and, if so, in step553 prioritization module 136 determines if all of the issue-creditcounters have overflowed. If, in step 553 prioritization module 136determines that all of the issue-credit counters have not overflowed,then in step 554 any overflowed issue-credit counter(s) incremented instep 550 are saturated and the corresponding overflow bit(s) are set.If, in step 553 prioritization module 136 determines that all of theissue-credit counters have overflowed, then in step 555 prioritizationmodule 136 reset all of the issue-credit counter overflow bits.

If, in step 552 prioritization module 136 determines that theincremented issue-credit counter did not overflow, then prioritizationmodule 136 proceeds directly to step 526. In other embodiments of thepresent invention, some threads or a thread type may be designated aslow priority and may be incremented by more than one for each issuedinstruction. Persons skilled in the art will appreciate that any systemconfigured to perform the method steps of FIG. 2, 4A, 4B, 4C, 5A, 5B, or5C, or their equivalents, is within the scope of the present invention.

The current invention involves new systems and methods for qualifyingbuffered current instructions to produce a set of qualified instructionsbased on qualification rules and prioritizing said qualifiedinstructions to produce prioritized instructions based on prioritizationrules to improve utilization of said different types of execution units.One or more of the prioritized instructions are then issued to executionunits for execution.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the invention.However, it will be apparent to one skilled in the art that specificdetails are not required in order to practice the invention. Thus, theforegoing descriptions of specific embodiments of the invention arepresented for purposes of illustration and description. They are notintended to be exhaustive or to limit the invention to the precise formsdisclosed; obviously, many modifications and variations are possible inview of the above teachings. The embodiments were chosen and describedin order to best explain the principles of the invention and itspractical applications, they thereby enable others skilled in the art tobest utilize the invention and various embodiments with variousmodifications as are suited to the particular use contemplated. Theforegoing description and drawings are, accordingly, to be regarded inan illustrative rather than a restrictive sense. The listing of steps inmethod claims do not imply performing the steps in any particular order,unless explicitly stated in the claim.

All trademarks are the respective property of their owners.

1. A multi-threaded processor, comprising: a plurality of differenttypes of execution units, at least one of each type of execution unitservicing a different class of operations; an instruction buffer that isconfigured to buffer a set of instructions for a plurality ofasynchronous threads, each instruction requiring an operation performedby one of each of the plurality of different types of execution units; amonitor unit coupled to the plurality of different types of executionunits and configured to monitor status of the plurality of differenttypes of execution units; and an instruction scheduler coupled to theinstruction buffer and the monitor unit, configured to receive inputfrom the instruction buffer and the monitor unit, and also configuredto: qualify at least a portion of the set of instructions for executionbased on a qualification rule stored in memory and the status of theplurality of different types of execution units to produce a set ofqualified instructions; wherein a remaining portion of the set ofinstructions are unqualified for execution based on the qualificationrule and the status of the plurality of different types of executionunits; prioritize the set of qualified instructions, and not theunqualified instructions, based on a prioritization rule to produceprioritized instructions for issue from said instruction buffer to theplurality of different types of execution units; and provide an outputrepresenting at least a portion of the prioritized instructions to beissued to at least a portion of the plurality of different types ofexecution units.
 2. The multi-threaded processor of claim 1, wherein theinstructions in the set of instructions each have an associated phaseidentifier and the set of qualified instructions includes instructionsthat are associated with a first phase identifier and excludes anyinstructions that are not associated with the first phase identifier,wherein the associated phase identifiers provide an indication to theinstruction scheduler of relative chronology of execution of therespective instructions.
 3. The multi-threaded processor of clam 2,wherein the set of qualified instructions includes instructions of afirst class that are configured to perform computation operations andinstructions of a second class that are configured to perform memoryaccess operations.
 4. The multi-threaded processor of claim 3, wherein aquantity of instructions of the second class is limited based on anamount of data returned for a memory access operation that a cache isconfigured to store.
 5. The multi-threaded processor of clam 3, whereina quantity of instructions of the second class is limited based onlocality of the memory access operations.
 6. The multi-threadedprocessor of claim 3, wherein the instruction scheduler is furtherconfigured to qualify instructions in the set of instructions that areassociated with the first phase identifier and wait to startqualification of instructions in the set of instructions that areassociated with the second phase identifier and of the second classuntil all memory access operations of the qualified instructions arecompleted.
 7. The multi-threaded processor of claim 1, wherein theprioritization rule specifies that the prioritized instructions areproduced based on issue-credit counters maintained for the plurality ofasynchronous threads.
 8. The multi-threaded processor of claim 7,wherein a portion of the issue-credit counters are incremented eachissue cycle when a prioritized instruction is issued from theinstruction buffer to reduce an issue priority of asynchronous threadscorresponding to the portion of issue-credit counters.
 9. Themulti-threaded processor of claim 1, wherein the status indicates that awrite-back conflict exists for a register specified by one of theplurality of asynchronous threads.
 10. The multi-threaded processor ofclaim 1, wherein said qualification rule specifies that an instructionwith write-back conflict is excluded from the set of qualifiedinstructions.
 11. The multi-threaded processor of claim 1, wherein thestatus indicates that threads are stalled in one or more of theplurality of different types of execution units.
 12. The multi-threadedprocessor of claim 1, wherein the instruction scheduler is furtherconfigured to repeat the prioritization of the set of qualifiedinstructions when the instruction buffer is refreshed with additionalinstructions.
 13. The multi-threaded processor of claim 1, wherein theinstruction scheduler is further configured to repeat the prioritizationof the set of qualified instructions when the status changes.
 14. Themulti-threaded processor of claim 1, further comprising a programmablerules engine configured to provide the qualification rule and to providethe prioritization rule.
 15. A method of operating a multi-threadedprocessor, comprising: buffering a set of instructions for asynchronousthreads, an instruction of an individual thread requiring one of aplurality of different types of execution units to service a class ofoperations; monitoring a status of the plurality of different types ofexecution units; qualifying at least a portion of the set of bufferedinstructions based on a qualification rule stored in memory and thestatus of the plurality of different types of execution units to producea set of qualified instructions; wherein a remaining portion of the setof buffered instructions are unqualified for execution based on thequalification rule and the status of the plurality of different types ofexecution units; prioritizing the qualified instructions, and not theunqualified instructions, to produce prioritized instructions based on aprioritization rule to improve utilization of the plurality of differenttypes of execution units; and for each issue cycle, issuing at least aportion of the prioritized instructions to the plurality of differenttypes of execution units for execution to produce issued instructions.16. The method of claim 15, wherein the instructions in the set ofinstructions each have an associated phase identifier and the set ofqualified instructions includes instructions that are associated with afirst phase identifier and excludes any instructions that are notassociated with the first phase identifier, wherein the associated phaseidentifiers provide an indication to the instruction scheduler ofrelative chronology of execution of the respective instructions.
 17. Themethod of clam 16, wherein the set of qualified instructions includesinstructions of a first class that are configured to perform computationoperations and instructions of a second class that are configured toperform memory access operations.
 18. The method of claim 17, wherein aquantity of instructions of the second class is limited based on anamount of data returned for a memory access operation that a cache isconfigured to store.
 19. The method of claim 18, wherein a quantity ofinstructions of the second class is limited based on locality of thememory access operations.
 20. The method of claim 18, wherein the stepof qualifying comprises: qualifying a portion of the bufferedinstructions that are associated with the first phase identifier; andwaiting to start qualification of another portion of the bufferedinstructions that are associated with the second phase identifier and ofthe second class until all memory access operations of the qualifiedinstructions are completed.